hostspot从jdk1.3就出来了，是sun的新一代jvm技术，主要目的是提高性能。实际应用中需要设置参数:
http://www.hashei.me/2009/05/tuning-the-sun-hotspot-jdk.html
回收器选择 JVM给了三种选择：串行收集器、并行收集器[吞吐量]、并发收集器[响应速度]，但是串行收集器只适用于小数据量的情况，所以这里的选择主要针对并行收集器和并发收集器。默认情况下，JDK5.0以前都是使用串行收集器，如果想使用其他收集器需要在启动时加入相应参数。JDK5.0以后，JVM会根据当前系统配置进行判断。
Continue reading java hotspot虚拟机
相关文章：
http://www.infoq.com/news/2011/02/agile-manifesto-10-years-series
http://www.infoq.com/cn/articles/personal-reflection-agile-ten-years-mellor
http://www.infoq.com/cn/articles/agile-teenage-crisis
虽然没有完完全全的按部就班的实施过某种敏捷方法，但是实际开发中确实与其中的思想不谋而合。我关注敏捷是由于CMM/CMMI这个东西以及大量的书籍中的观点似乎完全忽略了人以及编码这些要素。似乎软件的管理就是一堆堆的文档规范，众多的委员会骑在开发者的头上，然我看了很反感。难道文档编好了加上坐在办公椅上审视报告就能制造高质量的能work的软件？你们把实际做事的放哪里去了？这很自然地助长了官僚主义。
敏捷的观点是另一派的，我了解了后觉得符合实际，这些人是从下到上实践来的观点。
这一系列文章也反思了一下敏捷，我觉得：
CMM/CMMI还是敏捷方法有时都代表了某些人的利益，被过大的强调了；
敏捷和注重设计的方法本身应该不是对立的，其目的是让软件做好，只是各有侧重，而侧重并不是说抛弃对方的观点。重要的是结合实际情况来实施，没有可以照抄的成功，实事求是才是真理。
敏捷揭露了CMM/CMMI以及过度设计的反面，但是敏捷需要与其它观点共存！
Continue reading 【评】《敏捷宣言》10周年系列纪念文章
in Web前端 on web html css 前端
parentNode与parentElement同义，但是parentElement不是w3c规范的。这里主要讨论parentNode,offsetParent的区别：
offsetParent直接的将是影响元素位置的上级element，而parentElement与位置显示无关时dom中的上级element。 例如： <BODY> <div style="border: 1px solid black;position:absolute;"> <form> <input type="checkbox" id="cc"> </form> </div> 这个例子中，“cc”元素的offsetParent是div，如果去掉div的position属性，那么cc的offsetParent就会变为body。而parentElement一直都为form。 与此相关的还有offsetLeft和offsetTop两种属性，他们分别表示的是元素与offsetElement相对应的左侧和顶部距离。
另外的例子是table:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head></head> <script type="text/javascript"> window.onload = function(){ var el = document.getElementById('t2'); alert(el.offsetParent.id); }; </script> <body style="width: 1000px;"> <table id="tb"> <tr id="tr1"><td>t1</td></tr> <tr id="tr2"><td id="t2">t2</td></tr> </table> </body> </html>
可以看到td的offsetParent直接是table。
Continue reading parentNode，parentElement,offsetParent区别
If I were king, I'd banish JPEGs forever. JPEGs (pronounced "jay-pegs", and sometimes
spelled "JPGs") are the images you see all over the Web. Pictures stored as JPEGs are highly
compressed so they can be sent across the Internet faster. A picture that takes up 1 million bytes
can be squeezed down to 1/10th of that size if you turn it into a JPEG.
That would seem like a good idea. But it's actually a good idea gone bad. JPEGs are like
jigsaw puzzles that can never be put back together again right. Pieces will always be missing.
Every time you change a normal image into a JPEG, you lose part of the picture. You can never
get it back.
Remember that. I'll repeat it in a slightly different way to help out: When you create or save
a JPEG image, you lose part of the picture and you simply can't get it back, ever.
JPEGs are losers. They use what's called "lossy" compression. To squeeze an image's file
size down to something really small, the JPEG process tosses out 80 or 90 percent of the
information and fakes a lot of the rest. JPEGs take advantage of a trick of our eyesight. When we
see part of a small, familiar object, we automatically fill in the rest of it. JPEG processing breaks
an image up into a lot of small pieces and strips out a great deal of detail from each small piece.
A picture of your grandmother holding a shawl might show every detail ordinarily, but a
JPEG version might not show the way her fingers wrap around the fabric. Instead, they might be
depicted as areas of light and dark contrast that suggest an old woman's fingers. Your eyes and
brain do the rest. They turn what is only a suggestion of fingers into what your brain thinks is the
real thing.
This might be fine for some uses. But anyone who cares about digital photography should
consider JPEG as an enemy, not a friend. I'll bet you don't want that picture of grandma turned
into someone who looks sorta like grandma. You want the real thing.
JPEGs are one-way trips to the Badlands. Here's a familiar scenario. You have a normal,
non-compressed image. It's a Windows bitmap, maybe. (They're BMP files.) You open this
bitmap in your image-editing software and save it as a JPEG. The file is a lot smaller. You
celebrate a little. What a trick you just pulled!
Later, you realize you need to tweak the picture a little. Maybe you need to crop it. Maybe
you just want to make the colors more vivid.
So you open the JPEG in your image editor, make your changes and save the file as a JPEG
again.
The image you cropped or tweaked can't be the same as the one you started with. It was
already chopped up a lot by the JPEG processing, and when you saved it again it got chopped and
diced some more.
The PNG image format is the best thing since the wheel. PNG (pronounced "Ping," not "Pea
En Gee") means "Portable Network Graphics." When you save an image as a PNG, nothing gets
sliced or diced. A PNG is almost surely going to be smaller than a standard uncompressed image,
but it won't have any wounds. Cut it up all you want, slice it hither and yon. It will be all there,
even though it's smaller.
This miracle is actually just a trick of counting. We all do the same thing that PNGs do. We
learned this trick in grade school.
Here's how it works. Suppose you have a lot of sheep to count. You could count every
single one of them. Boring, right? Or you could count them by twos. Or maybe, if your eye is
quick, you could count them by threes.
Let's say you've done thismaybe you were trying the old technique of counting sheep to
get to sleep but it didn't workand now it's time to come up with the real number. If you counted
by twos, you just multiply your total by two. If you counted by threes, you multiply your number
by three.
Cool, right? You're storing something like 31 in your brain, but you can quickly turn that
into the real number62 or 93.
It's all shorthand. It's a perfectly sane mathematical trick. PNG uses the same approach. It
squeezes the data in an image file using tricks of math. Does the image have 61 pixels of nothing
but azure blue in one area? PNG stores that information as an instruction to make an azure blue
pixel 61 times. That takes up a lot less space than something like this: azure blue pixel, azure
blue pixel, azure blue pixel, ... you get the point.
The best part of this trick is the way it can be totally reversed without losses. the sky looks
exactly the way it was supposed to in the image with all those azure blue pixels. Storing
information by listing it as "1 pixel times 61" is the same as storing 61 pixels. Except that it
doesn't take up all that wasted space.
PNGs can't quite squeeze images as much as JPEGs can. There's no way to do it the lossless
way and get the same reduction in file size. But PNGs can cut an image down to anywhere from
one-half to one-third of its normal file size without any loss of data. Some images can be reduced
even more.
I took a few hundred uncompressed digital images and changed them from BMP to PNG to
find out how much space I could save. Most of them were 12 megabyte files stored as BMPs.
PNG squeezed these scans down to a little less than half their uncompressed size on average.
Some scans, especially ones that did not have much detail, compressed a lot more. But since I
started storing most of my digital images as PNGs, I've come to expect about a 50 percent to 60
percent savings in file space.
That might not seem like much compared to the way JPEG squeezes images, but it's a huge
bonus when you realize that PNG is harmless. Images are not changed in any way.
Good image viewers and editors know how to deal with PNG. If you don't see PNG listed as
an option in your image editor's "Save As" menu (under the "File" menu), you have old software
and should get a newer version.
If you don't have an image viewer of any kind, buy ACDSee, available for both Windows
PCs and Macintoshes.
1, shawl [ʃɔ:l]
n. 披肩,围巾
2, sorta ['sɔ:tə]
adv. 近似，有几分；可以说是
3, vivid ['vivid]
a. 生动的,栩栩如生的,鲜艳的
4, dice [dais]
vt. 切成方块
n. 骰子
vi. 掷骰子
5, sliced [slaist]
adj. （食物）已切成薄片的
v. 切成薄片；分配（slice的过去式和过去分词）
6, yon [jɔn]
adj. 那边的，彼处的
adv. 在那边，在远处
n. 那边，远处
pron. 彼处之人或物
7, chopped
v. 斩碎(开裂,中断,多变,开路)
8, sane [sein]
a. 神智健全的,理智的
9, azure
n. 天蓝色,碧空
a. 蔚蓝的
10, squeeze [skwi:z]
n. 压榨,挤
v. 紧握,挤
PNG原来是无损压缩位图
GIF也是无损压缩位图,只是由于压缩算法专利问题，逐渐被PNG取代。
http://zh.wikipedia.org/wiki/GIF
http://zh.wikipedia.org/wiki/PNG
Continue reading it-e-70 Why You should Not Use JPEGs for Image Storage
Why is image compression so important? Image files, in an uncompressed form, are very large.
And the Internet, especially for people using a 56k dialup modem, can be pretty slow.[1]This
combination could seriously limit one of the Web's most appreciated aspectsits ability to present
images easily.
JPEG (Joint Photographic Experts Group)compression is currently the best way to compress
PHOTOGRAPHIC IMAGES for the web. Other forms of image compression, including GIF and
PNG, are best used for other purposes on the web.
GIF (Graphics Interchange Format) is best used for graphics that have a limited color pallet
and large areas of flat tone, like cartoons or banners. Although it has several remarkable features,
such as transparency and the ability to present animated images, it is not well suited for the
presentation of continuous tone images, such as photographs, due to its limit of 256 colors.
PNG (Portable Network Graphics) is a relatively new format with a lot of potential but, until
all browsers can see images compressed in PNG form, it is not a good idea to use it.
JPEG, or JPG, is an evolving format that is universal in its use as a means of compressing
continuous tone photographs for speedy transmission over the Internet. Photographs compressed
using the JPEG format look good because JPEG supports millions of colors, so you can see the
gradation of tones.
A Bitmap is a simple series of pixels all stacked up. But the same image saved in GIF or
JPEG format uses less bytes to make up the file. How? Compression.
"Compression" is a computer term that represents a variety of mathematical formats used to
compress an image's byte size. Let's say you have an image where the upper right-hand corner
has four pixels all the same color. Why not find a way to make those four pixels into one?[2]That
would cut down the number of bytes by three-fourths, at least in the one corner. That's a
compression factor.
Bitmaps can be compressed to a point. The process is called "run-length encoding." Runs of
pixels that are all the same color are all combined into one pixel. [3]The longer the run of pixels,
the more compression. Bitmaps with little detail or color variance will really compress. Those
with a great deal of detail don't offer much in the way of compression. Bitmaps that use the
run-length encoding can carry either the common ".bmp" extension or ".rle". Another difference
between the two files is that the common Bitmap can accept 16 million different colors per pixel.
Saving the same image in run-length encoding knocks the bits-per-pixel down to 8. That locks
the level of color in at no more than 256.
So, why not create a single pixel when all of the colors are close? You could even lower the
number of colors available so that you would have a better chance of the pixels being close in
color. Good idea. The people at CompuServe felt the same way.
GIF, which stands for "Graphic Interchange Format," was first standardized in 1987 by
CompuServe, although the patent for the algorithm (mathematical formula) used to create GIF
compression actually belongs to Unisys. The first format of GIF used on the Web was called
GIF87a, representing its year and version. It saved images at 8 bits-per-pixel, capping the color
level at 256. That 8-bit level allowed the image to work across multiple server styles, including
CompuServe, TCP/IP.
CompuServe updated the GIF format in 1989 to include animation, transparency, and interlacing.
They called the new format, you guessed it: GIF89a.
There's no discernable difference between a basic (known as non-interlaced) GIF in 87 and
89 formats.
1, continuous tone
连续色调
2, gradation [ɡrə'deiʃən]
n. （色彩、颜色、次序、音调等的）渐变；分等级；（各种状态、性质等的）分阶段渐变；元音交替
3, discernable [di'sə:nəbl, -'zə:-]
adj. 可辨别的；可认识的
4, interlaced [,intə'leist]
a. 交织的,交错的
5, interlacing [intə(:)'leisiŋ]
n. 交错，隔行；隔行扫描
v. 交错，交织（interlace现在分词）
Continue reading it-e-69 Image Compression
in Web前端 on javascript web 前端
Continue reading 【转】再谈JavaScript的数据类型问题 淘宝网拥有国内最具商业价值的海量数据。截至当前，每天有超过30亿的店铺、商品浏览记录，10亿在线商品数，上千万的成交、收藏和评价数据。如何从这些数据中挖掘出真正的商业价值，进而帮助淘宝、商家进行企业的数据化运营，帮助消费者进行理性的购物决策，是淘宝数据平台与产品部的使命。 为此，我们进行了一系列数据产品的研发，比如为大家所熟知的量子统计、数据魔方和淘宝指数等。尽管从业务层面来讲，数据产品的研发难度并不高；但在 “海量”的限定下，数据产品的计算、存储和检索难度陡然上升。本文将以数据魔方为例，向大家介绍淘宝在海量数据产品技术架构方面的探索。 淘宝海量数据产品技术架构 数据产品的一个最大特点是数据的非实时写入，正因为如此，我们可以认为，在一定的时间段内，整个系统的数据是只读的。这为我们设计缓存奠定了非常重要的基础。 按照数据的流向来划分，我们把淘宝数据产品的技术架构分为五层（如图1所示），分别是数据源、计算层、存储层、查询层和产品层。位于架构顶端的是我们的数据来源层，这里有淘宝主站的用户、店铺、商品和交易等数据库，还有用户的浏览、搜索等行为日志等。这一系列的数据是数据产品最原始的生命力所在。 在数据源层实时产生的数据，通过淘宝自主研发的数据传输组件DataX、DbSync和Timetunnel准实时地传输到一个有1500个节点的 Hadoop[注：apache开源分布文件系统，主要用途搜索引擎。]集群上，这个集群我们称之为“云梯”，是计算层的主要组成部分。在“云梯”上，我们每天有大约40000个作业对1.5PB的原始数据按照产品需求进行不同的MapReduce[注：Google提出的概念，MapReduce是一种编程模型，用于大规模数据集（大于1TB）的并行运算。概念"Map（映射）"和"Reduce（化简）"，和他们的主要思想，都是从函数式编程语言里借来的，还有从矢量编程语言里借来的特性。他极大地方便了编程人员在不会分布式并行编程的情况下,将自己的程序运行在分布式系统上]计算。这一计算过程通常都能在凌晨两点之前完成。相对于前端产品看到的数据，这里的计算结果很可能是一个处于中间状态的结果，这往往是在数据冗余与前端计算之间做了适当平衡的结果。 不得不提的是，一些对实效性要求很高的数据，例如针对搜索词的统计数据，我们希望能尽快推送到数据产品前端。这种需求再采用“云梯”来计算效率将是比较低的，为此我们做了流式数据的实时计算平台，称之为“银河”。“银河”也是一个分布式系统，它接收来自TimeTunnel的实时消息，在内存中做实时计算，并把计算结果在尽可能短的时间内刷新到NoSQL存储设备中，供前端产品调用。 容易理解，“云梯”或者“银河”并不适合直接向产品提供实时的数据查询服务。这是因为，对于“云梯”来说，它的定位只是做离线计算的，无法支持较高的性能和并发需求；而对于“银河”而言，尽管所有的代码都掌握在我们手中，但要完整地将数据接收、实时计算、存储和查询等功能集成在一个分布式系统中，避免不了分层，最终仍然落到了目前的架构上。 为此，我们针对前端产品设计了专门的存储层。在这一层，我们有基于MySQL的分布式关系型数据库集群MyFOX和基于HBase的NoSQL存储集群Prom，在后面的文字中，我将重点介绍这两个集群的实现原理。除此之外，其他第三方的模块也被我们纳入存储层的范畴。 存储层异构模块的增多，对前端产品的使用带来了挑战。为此，我们设计了通用的数据中间层——glider——来屏蔽这个影响。glider以HTTP协议对外提供restful方式的接口。数据产品可以通过一个唯一的URL获取到它想要的数据。 以上是淘宝海量数据产品在技术架构方面的一个概括性的介绍，接下来我将重点从四个方面阐述数据魔方设计上的特点。 关系型数据库仍然是王道 关系型数据库（RDBMS）自20世纪70年代提出以来，在工业生产中得到了广泛的使用。经过三十多年的长足发展，诞生了一批优秀的数据库软件，例如Oracle、MySQL、DB2、Sybase和SQL Server等。 尽管相对于非关系型数据库而言，关系型数据库在分区容忍性（Tolerance to Network Partitions）方面存在劣势，但由于它强大的语义表达能力以及数据之间的关系表达能力，在数据产品中仍然占据着不可替代的作用。 淘宝数据产品选择MySQL的MyISAM引擎作为底层的数据存储引擎。在此基础上，为了应对海量数据，我们设计了分布式MySQL集群的查询代理层——MyFOX，使得分区对前端应用透明。 目前，存储在MyFOX中的统计结果数据已经达到10TB，占据着数据魔方总数据量的95%以上，并且正在以每天超过6亿的增量增长着（如图2所示）。这些数据被我们近似均匀地分布到20个MySQL节点上，在查询时，经由MyFOX透明地对外服务（如图3所示）。 值得一提的是，在MyFOX现有的20个节点中，并不是所有节点都是“平等”的。一般而言，数据产品的用户更多地只关心“最近几天”的数据，越早的数据，越容易被冷落。为此，出于硬件成本考虑，我们在这20个节点中分出了“热节点”和“冷节点”（如图4所示）。 顾名思义，“热节点”存放最新的、被访问频率较高的数据。对于这部分数据，我们希望能给用户提供尽可能快的查询速度，所以在硬盘方面，我们选择了每分钟15000转的SAS硬盘，按照一个节点两台机器来计算，单位数据的存储成本约为4.5W/TB。相对应地，“冷数据”我们选择了每分钟7500转的 SATA硬盘，单碟上能够存放更多的数据，存储成本约为1.6W/TB。 将冷热数据进行分离的另外一个好处是可以有效降低内存磁盘比。从图4可以看出，“热节点”上单机只有24GB内存，而磁盘装满大约有 1.8TB（300 * 12 * 0.5 / 1024），内存磁盘比约为4:300，远远低于MySQL服务器的一个合理值。内存磁盘比过低导致的后果是，总有一天，即使所有内存用完也存不下数据的索引了——这个时候，大量的查询请求都需要从磁盘中读取索引，效率大打折扣。 NoSQL是SQL的有益补充 在MyFOX出现之后，一切都看起来那么完美，开发人员甚至不会意识到MyFOX的存在，一条不用任何特殊修饰的SQL语句就可以满足需求。这个状态持续了很长一段时间，直到有一天，我们碰到了传统的关系型数据库无法解决的问题——全属性选择器（如图5所示）。 这是一个非常典型的例子。为了说明问题，我们仍然以关系型数据库的思路来描述。对于笔记本电脑这个类目，用户某一次查询所选择的过滤条件可能包括 “笔记本尺寸”、“笔记本定位”、“硬盘容量”等一系列属性（字段），并且在每个可能用在过滤条件的属性上，属性值的分布是极不均匀的。在图5中我们可以看到，笔记本电脑的尺寸这一属性有着10个枚举值，而“蓝牙功能”这个属性值是个布尔值，数据的筛选性非常差。 在用户所选择的过滤条件不确定的情况下，解决全属性问题的思路有两个：一个是穷举所有可能的过滤条件组合，在“云梯”上进行预先计算，存入数据库供查询；另一个是存储原始数据，在用户查询时根据过滤条件筛选出相应的记录进行现场计算。很明显，由于过滤条件的排列组合几乎是无法穷举的，第一种方案在现实中是不可取的；而第二种方案中，原始数据存储在什么地方？如果仍然用关系型数据库，那么你打算怎样为这个表建立索引？ 这一系列问题把我们引到了“创建定制化的存储、现场计算并提供查询服务的引擎”的思路上来，这就是Prometheus（如图6所示）。 从图6可以看出，我们选择了HBase作为Prom的底层存储引擎。之所以选择HBase，主要是因为它是建立在HDFS之上的，并且对于 MapReduce有良好的编程接口。尽管Prom是一个通用的、解决共性问题的服务框架，但在这里，我们仍然以全属性选择为例，来说明Prom的工作原理。这里的原始数据是前一天在淘宝上的交易明细，在HBase集群中，我们以属性对（属性与属性值的组合）作为row-key进行存储。而row-key 对应的值，我们设计了两个column-family，即存放交易ID列表的index字段和原始交易明细的data字段。在存储的时候，我们有意识地让每个字段中的每一个元素都是定长的，这是为了支持通过偏移量快速地找到相应记录，避免复杂的查找算法和磁盘的大量随机读取请求。 图7用一个典型的例子描述的Prom在提供查询服务时的工作原理，限于篇幅，这里不做详细描述。值得一提的是，Prom支持的计算并不仅限于求和 SUM运算，统计意义上的常用计算都是支持的。在现场计算方面，我们对Hbase进行了扩展，Prom要求每个节点返回的数据是已经经过“本地计算”的局部最优解，最终的全局最优解只是各个节点返回的局部最优解的一个简单汇总。很显然，这样的设计思路是要充分利用各个节点的并行计算能力，并且避免大量明细数据的网络传输开销。 用中间层隔离前后端 上文提到过，MyFOX和Prom为数据产品的不同需求提供了数据存储和底层查询的解决方案，但随之而来的问题是，各种异构的存储模块给前端产品的使用带来了很大的挑战。并且，前端产品的一个请求所需要的数据往往不可能只从一个模块获取。 举个例子，我们要在数据魔方中看昨天做热销的商品，首先从MyFOX中拿到一个热销排行榜的数据，但这里的“商品”只是一个ID，并没有ID所对应的商品描述、图片等数据。这个时候我们要从淘宝主站提供的接口中去获取这些数据，然后一一对应到热销排行榜中，最终呈现给用户。 有经验的读者一定可以想到，从本质上来讲，这就是广义上的异构“表”之间的JOIN操作。那么，谁来负责这个事情呢？很容易想到，在存储层与前端产品之间增加一个中间层，它负责各个异构“表”之间的数据JOIN和UNION等计算，并且隔离前端产品和后端存储，提供统一的数据查询服务。这个中间层就是glider（如图8所示）。 缓存是系统化的工程 除了起到隔离前后端以及异构“表”之间的数据整合的作用之外，glider的另外一个不容忽视的作用便是缓存管理。上文提到过，在特定的时间段内，我们认为数据产品中的数据是只读的，这是利用缓存来提高性能的理论基础。 在图8中我们看到，glider中存在两层缓存，分别是基于各个异构“表”（datasource）的二级缓存和整合之后基于独立请求的一级缓存。除此之外，各个异构“表”内部可能还存在自己的缓存机制。细心的读者一定注意到了图3中MyFOX的缓存设计，我们没有选择对汇总计算后的最终结果进行缓存，而是针对每个分片进行缓存，其目的在于提高缓存的命中率，并且降低数据的冗余度。 大量使用缓存的最大问题就是数据一致性问题。如何保证底层数据的变化在尽可能短的时间内体现给最终用户呢？这一定是一个系统化的工程，尤其对于分层较多的系统来说。 图9向我们展示了数据魔方在缓存控制方面的设计思路。用户的请求中一定是带了缓存控制的“命令”的，这包括URL中的query string，和 HTTP头中的“If-None-Match”信息。并且，这个缓存控制“命令”一定会经过层层传递，最终传递到底层存储的异构“表”模块。各异构“表” 除了返回各自的数据之外，还会返回各自的数据缓存过期时间（ttl），而glider最终输出的过期时间是各个异构“表”过期时间的最小值。这一过期时间也一定是从底层存储层层传递，最终通过HTTP头返回给用户浏览器的。 缓存系统不得不考虑的另一个问题是缓存穿透与失效时的雪崩效应。缓存穿透是指查询一个一定不存在的数据，由于缓存是不命中时被动写的，并且出于容错考虑，如果从存储层查不到数据则不写入缓存，这将导致这个存在的数据每次请求都要到存储层去查询，失去了缓存的意义。 有很多种方法可以有效地解决缓存穿透问题，最常见的则是采用布隆过滤器，将所有可能存在的数据哈希到一个足够大的bitmap中，一个一定不存在的数据会被这个bitmap拦截掉，从而避免了对底层存储系统的查询压力。在数据魔方里，我们采用了一个更为简单粗暴的方法，如果一个查询返回的数据为空（不管是数据不存在，还是系统故障），我们仍然把这个空结果进行缓存，但它的过期时间会很短，最长不超过五分钟。 缓存失效时的雪崩效应对底层系统的冲击非常可怕。遗憾的是，这个问题目前并没有很完美的解决方案。大多数系统设计者考虑用加锁或者队列的方式保证缓存的单线程（进程）写，从而避免失效时大量的并发请求落到底层存储系统上。在数据魔方中，我们设计的缓存过期机制理论上能够将各个客户端的数据失效时间均匀地分布在时间轴上，一定程度上能够避免缓存同时失效带来的雪崩效应。 结束语 正是基于本文所描述的架构特点，数据魔方目前已经能够提供压缩前80TB的数据存储空间，数据中间层glider支持每天4000万的查询请求，平均响应时间在28毫秒（6月1日数据），足以满足未来一段时间内的业务增长需求。 尽管如此，整个系统中仍然存在很多不完善的地方。[这句话真是太熟了]一个典型的例子莫过于各个分层之间使用短连接模式的HTTP协议进行通信。这样的策略直接导致在流量高峰期单机的TCP连接数非常高。所以说，一个良好的架构固然能够在很大程度上降低开发和维护的成本，但它自身一定是随着数据量和流量的变化而不断变化的。我相信，过不了几年，淘宝数据产品的技术架构一定会是另外的样子。 Continue reading 【转+评】淘宝数据魔方技术架构解析 Of course you can encrypt a graphics file. After all, most encryption algorithms don't care n. 零,暗号 Continue reading it-e-68 Can Graphics Files Be Enctypted For most types of graphics file formats currently available the answer is "no". A virus (or for information, or display other graphics and video information. And such multimedia display 1, catastrophe [kə'tæstrəfi] 2, sinister ['sinistə] 3, exorcise ['eksɔ:saiz] 4, coincidentally [kəu,insi'dentli] Continue reading it-e-67 Can Graphics Files Be Infected with a Virus An image is digitized to convert it to a form that can be stored in a computer's memory or on some form of storage media such as a hard disk or CD-ROM. This digitization procedure can be An image which has been captured in poor lighting conditions, and shows a continuous The example below demonstrates how one could go about extracting measurements from an frame grabber board 帧中继访问设备 1, corrugated ['kɔrəgeitid] 2, dilation [dai'leiʃən, di-] 3, subtract [səb'trækt] 4, segmenting 5, thresholding 6, watershed ['wɔ:təʃed, 'wɔ-] Continue reading it-e-66 Introduction to Digital Image Processing 【转+评】淘宝数据魔方技术架构解析
程序员原文:http://www.programmer.com.cn/7578/
读着很像论文体，故此收藏。其中很多概念可以先读读百科hadoop http://baike.baidu.com/view/908354.htm 和 http://www.infoq.com/cn/articles/hadoop-intro
it-e-68 Can Graphics Files Be Enctypted
about the intellectual content of a file. All they chew on is a series of byte values. Therefore,
most any encryption program that works on ordinary text files will work on graphics files as well.
Why would you want to encrypt a graphics file? Mostly to control who can view its contents.
You can invent a proprietary file format and that might slow a file format hack down for, say,
five or ten minutes. You could add a proprietary data compression scheme, possibly a twisted
variation of an already public algorithm. But there are so many people out there with nothing
better to do than hack at unknown data formats that your data would probably be exposed in little
time. But suppose we top off all this effort by encrypting the graphics file itself as we would an
ordinary text file. Would your data then be safe?
Realize that an encrypted graphics file still might not be very secure. For every data
encryption algorithm there exists at least one method of getting around it, although it may take
hundreds of computers and many years to fully employ and execute that method!
For example, one of the more popular methods used to encrypt data is the Vernam or XOR
cipher. This cipher Exclusive ORs the plain-text data with a single, random, fixed-length key.
The longer the key the harder it is to break the cipher. A totally random key the length of your
data is impossible to break. Shorter and less-random keys are easier to break.
XOR is very simple and fast, which is a must for a graphics file translators/viewers that
must decrypt a file on the fly. A problem, however, is that most graphics files contain fixed size
headers which vary only slightly in content from file to file. If you knew the approximate
contents of the header of an encrypted file you could XOR a "decrypted" header with the
encrypted file and possibly produce the key used to encrypt the file. A short key might be very
easily discovered in this way.
If you wish to use a public key/private key encryption method, then storing the public key in
the file format header (usually as a 4-byte field) and only encrypting the image data would be the
way to go. The SMPTE DPX file format supports such an encryption feature.
If you really need to make the contents of a graphics file secure, then I'd suggest not only
using some form of data encryption, but also create an unconventional and proprietary file format
and do not publish its format specification.cipher ['saifə]
it-e-67 Can Graphics Files Be Infected with a Virus
worm, Trojan horse, and so forth) is fundamentally a collection of code (that is, a program) that
contains instructions which are executed by a CPU. Most graphics files, however, contain only
static data and no executable code. The code that reads, writes, and displays graphics data is
found in translation and display programs and not in the graphics files themselves. If reading or
writing a graphics file caused a system malfunction is it most likely the fault of the program
reading the file and not of the graphics file data itself.
With the introduction of multimedia we have seen new formats appear, and modifications to
older formats made, that allow executable instructions to be stored within a file format. These
instructions are used to direct multimedia applications to play sounds or music, prompt the user
programs may perform these functions by interfacing with their environment via an API, or by
direct interaction with the operating system. One might also imagine a truly object-oriented
graphics file as containing the code required to read, write, and display itself.
Once again, any catastrophes that result from using these multimedia application is most
like the result of unfound bugs in the software and not some sinister instructions in the graphics
file data. Such "logic bombs" are typically exorcised through the use of testing using a wide
variety of different image files for test cases.
If you have a virus scanning program that indicates a specific graphics file is infected by
virus, then it is very possible that the file coincidentally contains a byte pattern that the scanning
programming recognizes as a key byte signature identifying a virus. Contact the author (or even
read the documentation!) of the virus scanning program to discuss the probability of the
mis-identification of a clean file as being infected by a virus. Save the graphics file, as the author
will most likely wish to examine it as well.
If you suspect a graphics file to be at the heart of a virus problem you are experiencing, then
also consider the possibility that the graphics file's transport mechanism (floppy disk, tape or
shell archive file, compressed archive file, and so forth) might be the original source of the virus
and not the graphics file itself.
n. 大灾难；大祸；惨败
a. 不吉利的,凶恶的,左边的
vt. 驱邪；除怪
adv. 巧合地；一致地 it-e-66 Introduction to Digital Image Processing
done by a scanner, or by a video camera connected to a frame grabber board in a computer. Once an
image has been digitized, it can be operated upon by various image processing operations.
Image processing operations can be roughly divided into three major categories, Image
Compression, Image Enhancement and Restoration, and Measurement Extraction[提取尺寸]. Image
compression is familiar to most people. [1]It involves reducing the amount of memory needed to
store a digital image.
Image defects which could be caused by the digitization process or by faults in the imaging
set-up (for example, bad lighting) can be corrected using Image Enhancement techniques. Once
the image is in good condition, the Measurement Extraction operations can be used to obtain
useful information from the image.
Some examples of Image Enhancement and Measurement Extraction are given below. The
examples shown all operate on 256 grey-scale images. This means that each pixel in the image is
stored as a number between 0 to 255, where 0 represents a black pixel, 255 represents a white
pixel and values in-between represent shades of grey. These operations can be extended to
operate on colour images.
The examples below represent only a few of the many techniques available for operating on
images. Details about the inner workings of the operations have not been given.
Image Enhancement and Restoration
The image at the left of Figure 1 has been corrupted 
by noise during the digitization process. The "clean"
image at the right of Figure 1 was obtained by applying a
median filter [中值滤波器]to the image.
An image with poor contrast, such as the one at
the left of Figure 2, can be improved by adjusting the
image histogram to produce the image shown at the
right of Figure 2.
The image at the top left of Figure 3 has a corrugated effect due to a fault in the acquisition
process. This can be removed by doing a 2-dimensional Fast-Fourier Transform[快速傅里叶变换] on the image
(top right of Figure 3), removing the bright spots (bottom left of Figure 3), and finally doing an
inverse Fast Fourier Transform to return to the original image without the corrugated background
(bottom right of Figure 3).
change in the background brightness across the image (top left of Figure 4) can be corrected
using the following procedure. First remove the foreground objects by applying a 25 by 25
greyscale dilation operation (top right of Figure 4). Then subtract the original image from the
background image (bottom left of Figure 4). Finally invert the colors and improve the contrast by
adjusting the image histogram (bottom right of Figure 4).
image. The image at the top left of Figure 5 shows some objects. The aim is to extract information
about the distribution of the sizes (visible areas) of the objects. The first step involves segmenting
the image to separate the objects of interest from the background. This usually involves
thresholding the image, which is done by setting the values of pixels above a certain threshold value
to white, and all the others to black (top right of Figure 5). Because the objects touch, thresholding
at a level which includes the full surface of all the objects does not show separate objects. This
problem is solved by performing a watershed separation on the image (lower left of Figure 5). The
image at the lower right of Figure 5 shows the result of performing a logical AND of the two
images at the left of Figure 5. This shows the effect that the watershed separation has on touching
objects in the original image. Finally, some measurements can be extracted from the image.
a. 缩成皱纹的,使起波状的
n. 扩张，扩大；膨胀；详述
v. 减去,扣掉,减少
n. 分段
n. 阈值转换法；域值
n. （美）流域；分水岭；集水区；转折点
adj. 标志转折点的
